fix(rag): harden GPT-5.6 answer generation#626
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR upgrades OpenAI routing to GPT-5.6, adds structured Responses API parsing, safety identifiers, cache fingerprinting, and stricter error handling. It replaces provisional answer streaming with validated final-only SSE, strengthens RAG fallbacks and quality gates, centralizes request validation, and updates tests and operational documentation. ChangesOpenAI runtime and workload routing
Validated final-only answer streaming
RAG classification, fallback, and quality gates
Operational configuration and validation support
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0136ba348b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
# Conflicts: # docs/branch-review-ledger.md # src/app/api/answer/stream/route.ts # src/components/ClinicalDashboard.tsx # src/lib/clinical-search.ts # src/lib/rag.ts # tests/clinical-search.test.ts
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f83d7d49b3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex resolve actionable Codex review findings for this pull request and current head using the repository instructions. This is the pull request's single automatic repair pass: do not perform a fresh review, create new standalone findings, or request another review. Work only the existing unresolved Codex threads on the current head. Always fix P0 and P1 findings. For P2 and lower findings, fix only clear, scoped, low-risk issues; otherwise disposition them with a concise reason. After fixing or dispositioning a thread, reply in that thread with as the first line, followed by a concise summary; that marker authorizes the workflow to close that exact thread. If human input or new authorization is required, do not use the marker and leave the thread open with the blocker. Finish only after every actionable thread is fixed or dispositioned and closed, or explicitly left open for a human decision. Do not update the branch from main, address unrelated reviews, broaden scope, or create more than one scoped fix commit. Do not use external APIs, paid services, credentials, dependency changes, or broad refactors unless explicitly authorized. Add targeted tests where behavior changes and run the narrowest relevant validation. |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
tests/answer-stream-contract.test.ts (1)
15-22: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueGuardrail test is coupled to exact source text, not behavior.
Asserting on raw substrings (
StreamingAnswerPreview,event === "token", etc.) protects against literal reintroduction of those tokens, but a harmless rename or restructuring (e.g. switch statement instead of===) would silently defeat the guard while the underlying regression it protects against could still resurface under different code. Consider a behavior-level test (render the dashboard, feed atoken/revisingSSE event, assert no provisional text appears) for a more durable guarantee.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/answer-stream-contract.test.ts` around lines 15 - 22, Replace the raw source-string assertions in the “has no provisional answer rendering path in the dashboard” test with a behavior-level test that renders ClinicalDashboard, feeds token and revising SSE events, and verifies no provisional answer text is displayed. Keep the test focused on preventing provisional rendering rather than specific implementation names or event-handling syntax.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/eval-answer-quality.ts`:
- Line 126: Remove the owner-scoped answer preview from the JSON object
assembled in the evaluation output so --json omits answer content by default. If
debugging access is required, expose it only through a separate explicit opt-in
path with redaction, without retaining private corpus-derived text in standard
logs or artifacts.
In `@src/lib/rag-answer-text.ts`:
- Around line 293-306: Broaden the unit allowlist in bareDoseFigurePattern to
exempt valid dose phrases using common units such as tablets and mmol, while
preserving the existing numeric and boundary matching behavior. Update only this
pattern so complete phrases like “maximum dose is 2 tablets” and “dose is 10
mmol” are not classified as artifacts.
In `@src/lib/rag-claim-support.ts`:
- Around line 397-401: Centralize the source_backed_review_fallback routing
token used by the claim-support gate and its producers. Define or reuse a shared
constant/helper alongside the existing routing-reason definitions in rag.ts,
then update sourceBackedReviewAnswer in the claim-support flow and all fallback
producers to reference it instead of duplicating the literal token.
In `@src/lib/rag.ts`:
- Around line 1223-1237: Update queryClassifierParseSchema to remove the
max-length, max-item, minimum, and maximum constraints from confidence, reasons,
and expandedTerms so Structured Outputs accepts the schema. Preserve their basic
types, then validate the corresponding bounds after parsing in the classifier
flow.
---
Nitpick comments:
In `@tests/answer-stream-contract.test.ts`:
- Around line 15-22: Replace the raw source-string assertions in the “has no
provisional answer rendering path in the dashboard” test with a behavior-level
test that renders ClinicalDashboard, feeds token and revising SSE events, and
verifies no provisional answer text is displayed. Keep the test focused on
preventing provisional rendering rather than specific implementation names or
event-handling syntax.
🪄 Autofix (Beta)
❌ Autofix failed (check again to retry)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 08dbbc5e-eaee-4da1-a94d-1468ed962977
📒 Files selected for processing (55)
.env.exampledocs/branch-review-ledger.mddocs/capacity-review.mddocs/codex-prompt-playbook.mddocs/openai-cross-border-basis.mddocs/openai-rag-operations.mddocs/privacy-impact-assessment.mddocs/search-rag-master-context.mddocs/worker-deploy-runbook.mdscripts/check-client-bundle-secrets.mjsscripts/eval-answer-quality.tsscripts/production-readiness.tssrc/app/api/answer/route.tssrc/app/api/answer/stream/route.tssrc/app/api/documents/[id]/table-facts/route.tssrc/components/ClinicalDashboard.tsxsrc/lib/answer-stream-contract.tssrc/lib/answer-stream-extractor.tssrc/lib/document-enrichment.tssrc/lib/env.tssrc/lib/eval-document-matching.tssrc/lib/model-index-extraction.tssrc/lib/openai.tssrc/lib/rag-answer-text.tssrc/lib/rag-cache.tssrc/lib/rag-claim-support.tssrc/lib/rag-eval-cases.tssrc/lib/rag-extractive-answer.tssrc/lib/rag-versioning.tssrc/lib/rag.tssrc/lib/sse-heartbeat.tssrc/lib/types.tssrc/lib/validation/answer-request.tstests/answer-progress-ui-smoke.spec.tstests/answer-responsiveness-gate.test.tstests/answer-stream-contract.test.tstests/answer-stream-extractor.test.tstests/client-secret-surface.test.tstests/corpus-grounding.test.tstests/document-enrichment.test.tstests/document-mutation-routes.test.tstests/model-index-extraction.test.tstests/openai-cache.test.tstests/openai-error-mapping.test.tstests/openai-safety-identifier.test.tstests/private-access-routes.test.tstests/private-rag-access.test.tstests/rag-answer-fallback.test.tstests/rag-answer-text.test.tstests/rag-cache-utils.test.tstests/rag-claim-support.test.tstests/rag-classifier-memo.test.tstests/rag-generation-fingerprint.test.tstests/rag-offline-answer.test.tstests/ui-smoke.spec.ts
💤 Files with no reviewable changes (2)
- tests/answer-stream-extractor.test.ts
- src/lib/answer-stream-extractor.ts
Remove clinical answer previews from evaluation output, keep valid dose evidence eligible, and validate classifier bounds after Structured Outputs parsing. Verified with focused tests, offline RAG, verify:cheap, and the production build.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d8bebf1039
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/lib/rag-extractive-answer.ts (1)
1680-1688: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winCount distinct cited documents for broad-coverage gating.
src/lib/rag-extractive-answer.ts:1686 Two citations from the same document can still satisfy this check; compare distinct citeddocument_ids instead ofanswer.citations.length, and add a regression test for duplicate-document citations.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/rag-extractive-answer.ts` around lines 1680 - 1688, Update the broad-coverage gating condition near broadDocumentCoverageRequested to compare the number of distinct cited document_id values against the required citation threshold, rather than answer.citations.length; preserve the existing source-availability check and add a regression test covering duplicate citations from one document.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lib/rag-extractive-answer.ts`:
- Around line 381-388: Update hasMaximumDoseEvidence to recognize spelled-out
dose units, including milligram(s) and the other long-form units already
supported by the dose parser, while preserving existing numeric maximum-dose
patterns. Add regression coverage for wording such as “must not exceed 20
milligrams” and the additional supported long-form units.
---
Outside diff comments:
In `@src/lib/rag-extractive-answer.ts`:
- Around line 1680-1688: Update the broad-coverage gating condition near
broadDocumentCoverageRequested to compare the number of distinct cited
document_id values against the required citation threshold, rather than
answer.citations.length; preserve the existing source-availability check and add
a regression test covering duplicate citations from one document.
🪄 Autofix (Beta)
❌ Autofix failed (check again to retry)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: f4a19637-b9da-4ac9-924a-6cafd46c221f
📒 Files selected for processing (11)
docs/branch-review-ledger.mdscripts/eval-answer-quality.tssrc/lib/rag-answer-text.tssrc/lib/rag-claim-support.tssrc/lib/rag-extractive-answer.tssrc/lib/rag-routing.tssrc/lib/rag.tstests/answer-stream-contract.test.tstests/extractive-answer-formatting.test.tstests/rag-answer-text.test.tstests/rag-classifier-memo.test.ts
💤 Files with no reviewable changes (1)
- tests/answer-stream-contract.test.ts
🚧 Files skipped from review as they are similar to previous changes (6)
- tests/rag-answer-text.test.ts
- docs/branch-review-ledger.md
- tests/rag-classifier-memo.test.ts
- src/lib/rag-answer-text.ts
- src/lib/rag-claim-support.ts
- src/lib/rag.ts
|
Outstanding provider gate completed post-merge. PR #626 was already squash-merged as
No code improvement was required by the result, so no follow-up commit or push was created. Later main commits through |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1c6c780f30
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Preserve retryable empty max-token responses and accept equivalent maximum-dose wording with long-form units. Verified with focused Vitest, offline RAG, verify:cheap, production readiness, and a production build.
Preserve the shared answer request and final-only SSE contracts while incorporating response-bound feedback tokens from main. Post-merge focused tests, offline RAG, verify:cheap, and production build passed.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0301d59ab8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Caution Failed to replace (edit) comment. This is likely due to insufficient permissions or the comment being deleted. Error details |
Recognize not-to-exceed and not-more-than maximum-dose evidence. Focused Vitest, typecheck, and the offline RAG contract passed.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f2652021c1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lib/rag-extractive-answer.ts`:
- Around line 354-365: Update maximum-dose evidence handling in
doseIntentEvidencePattern and the related
hasMaximumDoseEvidence/factKindForSentence flow so bare “max” or “maximum” no
longer qualifies as dose evidence; require those terms to be paired with “dose”
or a clinical dose value while preserving existing equivalent numeric
maximum-dose phrases. Add the smallest targeted regression asserting that
“Lithium has a maximum treatment duration of two years.” returns false, then run
the narrowest relevant validation.
🪄 Autofix (Beta)
❌ Autofix failed (check again to retry)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 6f1bc3ab-e44b-41e7-8f2d-f5ca558ec4a0
📒 Files selected for processing (13)
docs/branch-review-ledger.mdsrc/app/api/answer/route.tssrc/app/api/answer/stream/route.tssrc/components/ClinicalDashboard.tsxsrc/lib/openai.tssrc/lib/rag-answer-text.tssrc/lib/rag-extractive-answer.tssrc/lib/types.tstests/extractive-answer-formatting.test.tstests/openai-cache.test.tstests/private-access-routes.test.tstests/rag-answer-fallback.test.tstests/rag-answer-text.test.ts
🚧 Files skipped from review as they are similar to previous changes (10)
- src/lib/types.ts
- tests/rag-answer-text.test.ts
- tests/private-access-routes.test.ts
- src/app/api/answer/route.ts
- tests/rag-answer-fallback.test.ts
- src/lib/rag-answer-text.ts
- src/app/api/answer/stream/route.ts
- tests/openai-cache.test.ts
- src/lib/openai.ts
- src/components/ClinicalDashboard.tsx
|
@copilot resolve the merge conflicts on this branch. |
|
Note Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it. Autofix skipped. No unresolved CodeRabbit review comments with fix instructions found. |
|
Note Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it. Autofix skipped. No unresolved CodeRabbit review comments with fix instructions found. |
|
Note Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it. Autofix skipped. No unresolved CodeRabbit review comments with fix instructions found. |
|
Note Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it. Autofix skipped. No unresolved CodeRabbit review comments with fix instructions found. |
Head branch was pushed to by a user without write access
Head branch was pushed to by a user without write access
|
@copilot fix all failing CI tests |
Head branch was pushed to by a user without write access
…sources.ts (#641) Audit 2026-07-13 deferred item D3, rag.ts decomposition phase 1 (finding 11). Zero behavior change: 1,014 lines moved verbatim (script-extracted with boundary guards and per-symbol assertions), nothing rewritten. Moved: the versioned-RPC plumbing (callVersionedRetrievalRpc, legacy owner/public merge, RPC error + variant-fanout telemetry), the lexical chunk / document-lookup / table-fact / embedding-field / index-unit candidate searchers, the chunk hydrators for memory cards and signal matches, and the memory-boost merge step. rag.ts (5,779 -> ~4,790 lines) imports the surfaces it orchestrates and re-exports callVersionedRetrievalRpc, loadChunksForMemoryCards, and loadChunksForSignalMatches so existing test imports are untouched. The extracted module depends only on non-rag lib modules - no import cycle. Verified: typecheck + lint clean, eval:rag:offline green (279 contract tests), verify:cheap green (2,301 tests), verify:pr-local green (build + client-bundle scan). The extracted module is byte-identical whether generated from the pre- or post-#626 base (the hardening PR touched no moved range). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Summary
Verification
7e4b535fc53fe61ace561facdb8c7a224c18d86fnpm run verify:cheap: runtime/policy/lint/typecheck passed; 239 test files passed, 1 skipped; 2,248 tests passed, 1 skipped.npm run eval:rag:offline: 36 fixture-schema cases and 279/279 tests passed.npm run verify:ui: 158/158 Chromium checks passed.7e4b535fPR requiredpassed.release-browser-matrixwere skipped by workflow scope.npm run verify:release9f4dc13044f3403c9a977a18cecb0c537ca9fd08.npm run check:production-readinessOPENAI_SAFETY_IDENTIFIER_SECRET.npm run check:deployment-readinessClinical Governance Preflight
Clinical KB Database(sjrfecxgysukkwxsowpy).Notes
85137cde853550b59dd67dea7e0d12bb412e6692on 2026-07-14 before the final provider gate was rerun.9f4dc1304; later main commits through281d3e0cdchanged only documentation, Therapy Compass mockups, and document-viewer code, not the validated OpenAI/RAG path.OPENAI_SAFETY_IDENTIFIER_SECRETremains an operational/privacy follow-up; authenticated Responses requests omit that optional pseudonymous identifier until it is configured.Summary by CodeRabbit
New Features
Bug Fixes
User Experience
Documentation